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ABSTRACT 



The leadership role of the Office of Educational Research 
and Improvement (OERI) in developing a System of Expert Panels to identify 
and share promising and exemplary products, programs, and practices is 
discussed. The System of Expert Panels should enable the Federal government 
to help the public learn about replicable research and development (R&D) 
products and programs and to help funders and producers of R&D maximize their 
investments. Such a system should be more than a collection of individual 
expert panels. It should be led from the inside by a Federal Coordination 
Unit of experts from Department of Education offices. A‘ middle circle then 
would connect the individual expert panels in a comprehensive array of topic 
areas. An outside circle then would represent the consumer-oriented 
dissemination and evaluation providers. This outer circle would be composed 
of existing dissemination and evaluation entities such as ERIC and the 
Regional Educational Laboratories. Principles that guide OERI 1 s role in 
dissemination are: (1) the Federal role should be f acili tative ; (2) there 

should be a combined focus on evaluation and dissemination; (3) the System 
should help create a deliberate, effective, and continuous approach to 
finding gaps that merit R&D support; (4) national dissemination efforts 
should encourage selection and use of the best of educational R&D; (5) 
participants will increase their commitment to using research, development, 
and evaluation to improve education; and (6) the System of Expert Panels must 
adapt to face challenges such as finding financial support. (Contains 25 
references.) ( SLD) 
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An Overview of OERI Efforts to Develop A System to Designate and 
Disseminate Promising and Exemplary Products , Programs and Practices 

by Susan Klein 1 

Office of Reform Assistance and Dissemination, Knowledge Applications Division, 
Office of Educational Research and Improvement, U.S. Department of Education 

March 19, 1997 

As with many systems under design or even those with an extensive operational history 
such as the National Diffusion Network, different individuals have different perspectives 
about what has been or should be done to help find and share the best replicable R&D 
models with potential users. As researchers we want perspectives to be informed by good 
analytic insights which include learning from related activities. We also understand the 
value of consistent sets of principles and assumptions about what is needed and how to 
accomplish the goals. 

This paper describes my views about OERI’s leadership role in developing a System of 
Expert Panels to identify and share promising and exemplary products, programs and 
practices. My ideas are based on what is covered by the 1994 OERI reauthorization 
legislation 2 as well as research that I and my colleagues have been doing since the mid 
1970s on how federal offices learn about and share the best (Klein, 1993, 1996; Klein & 
Gwaltney, 1991; LaFollette, 1992). These ideas are also described in an article that 
should be published in the Educational Researcher soon on “A System of Expert Panels 
and Design Competitions: Complementary Federal Approaches to Find, Develop and 
Share Promising and Exemplary Products and Programs.” This article responds to Bob 
Slavin’s article on “Design Competitions: A Proposal for a New Federal Role in 
Educational Research and Development for the Jan / Feb. 1997 Educational Researcher. 

Background and Purposes: Under its 1994 Reauthorization (Title IX of Goals 2000: Educate 
America Act), OERI has been given responsibility for: 



This paper is prepared for the Annual Meeting of the American Educational Research Association in Chicago, 
Symposium 5.26 “Learning from Consumer-Oriented Review Efforts to Designate Promising and Exemplary Products, 
Programs and Practices” on March 24, 1997. It is based on a longer paper “A System of Expert Panels and Design 
Competitions: Complementary Federal Approaches to Find, Develop and Share Promising and Exemplary Products and 
Programs,” that has been cleared by the Department of Education for publication in the Educational Researcher . This shorter 
paper and the presentation upon which it is to be based are intended to promote the exchange of ideas among researchers and 
policy makers. The views are those of the author and no official support by the U.S. Department of Education is intended or 
should be inferred. 

2 

I was fortunate in being able to advise Congressman Major Owens and his Subcommittee staff as they prepared this 
legislation. 
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• developing standards to designate promising and exemplary products, programs and 
practices. Draft regulations for these standards published June 3, 1996 were revised in 
response to the public comment and will be published in draft again for additional public 
comment. 

• establishing a system of expert panels to make recommendations on these designations to the 
Secretary of Education. Two pilot panels in the areas of mathematics and science, and 
gender equity were appointed in 1996. 

• coordinating dissemination activities and programs not just within OERI, but with other parts 
of the Department, other agencies concerned with education, and with associations and other 
levels of government, such as state education agencies and local school districts. 

The System of Expert Panels should enable the federal government to: 

1) help the public learn about the comparative advantages of what exists among the many 
available replicable R&D-based products, programs, practices and policies 3 based on an 
understanding of their relative merits 4 ; 

2) help federal and other funders and producers of R&D-based resources maximize their 
investments by providing support for further evaluation, improvement and dissemination of 
existing promising and exemplary R&D-based resources and by identifying gaps which may help 
indicate areas where new products and programs are needed. Other expected benefits include 
providing practical reasons for educators (and in some cases students) to improve their 
evaluation skills related to the use of evaluation information in making decisions on instructional 
tools such as programs or products. Participants in System activities will also be able to 
contribute to the development of new methods to combine evaluation and dissemination 
functions to better serve education consumers. 

This System is congruent with the Department of Education’s strategic plan (1994) to better 
serve its customers, but it is not limited to reviewing models that have been developed with 
previous federal funding. It builds on, but differs from, the Joint Dissemination Review Panel 



3 These replicable R&D-based resources: 1) can be used outside of the original development site, 2) 
are based on principles from educational research and 3) have some evaluation evidence on their positive 
impact. In the June 3, 1996 draft standards and in this paper, “program” will often be used to refer to all 
these R&D-based models or resources. The OERI legislation includes research findings in this list, but 
instead of having expert panels designate promising or exemplary research studies, syntheses and 
interpretive papers or informational videos, it is most likely that research findings or principles will be 
used indirectly by including them in the criteria established by each panel. 

4 Merit is determined by expert panel reviews using criteria under the four categories in the OERI 
standards for designating promising and exemplary programs. These categories are: evidence of 
effectiveness/ success, quality, educational significance, and usefulness to others. (See the sidebar.) 
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and its successor the Program Effectiveness Panel used by the Department’s National Diffusion 
Network (NDN) which is no longer funded . 5 

Structure and Operational Principles: 

Structurally this Federally Led Nationwide System Includes More Than Expert Panels. 

It is important to develop a well understood and articulated nationwide system that has clear 
connections to other complementary systems, such as the National Education Dissemination 
System (NEDS) and the even more loosely configured R&D production system. Ideally, this 
System would be logical, inclusive and dynamic. As explained later, the federal role would be 
facilitative and collaborative, rather than regulatory. Experience with the development of the 
Gender Equity Expert panel shows that thinking in new ways is a challenge for many of us on 
even simple issues such as public acknowledgment about who has submitted what. The 
community of panel members and their advisers need to decide if they want all submissions to be 
known publicly and applauded and helped by the community or if they want them to be 
confidential until positive decisions are made by the Secretary of Education about their 
promising or exemplary status. 

The System of Expert Panels should be more than a collection of individual expert panels. The 
“Wheel Diagram of the Key Elements of the System of Expert Panels” shows that the System 
would be led and coordinated from the inside hub circle by a Federal Coordination Unit (FCU) 
of experts from various Department of Education Offices and other agencies involved in 
education activities to identify and share the best from R&D. This FCU would include agency 
representatives such as federal staff liaisons with current or potential expert panels. 

The middle circle connects the individual expert panels (P) in a comprehensive array of topic 
areas. Aside from the pilot Expert Panel in Mathematics and Science Education, other 
curriculum areas such as reading may be included. Among others with an equity focus, 
special education may join the Gender Equity Expert Panel. Other expert panels may 
focus on topic areas such as school-wide models, safe and drug-free schools, uses of 
education technology, and teacher education. Where feasible, each topic focused expert 
panel would have liaisons from federal offices who would be in a position to use money 



5 The National Diffusion Network (NDN) established in 1974 and last funded in 1995 provided a 
systematic procedure to identify and disseminate programs that had evidence of effectiveness. The 
System of Expert Panels builds on the NDN focus on reviewing submissions to judge effectiveness of 
replicable programs. But it goes beyond NDN in many ways, such as including three additional criteria 
categories and the deliberate use of experts, criteria and evidence appropriate for specific topic areas. 
NDN was a self-contained general purpose review and dissemination system with a review panel, ED 
funding of many of those approved by the panel as Developers/Demonstrators (to provide 
implementation assistance to adopters) and State Facilitators (to encourage the use of all approved 
programs). 
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and knowledge from their programs to contribute to and build on panel work. These 
federal liaisons would also advise on the selection of panel members who would be approved by 
their leaders. This circle also contains a standing panel and a democratic governance and 
coordination structure for the expert panels who would represent their topic-focused constituency 
groups. 

The outside circle represents the consumer-oriented dissemination and evaluation providers 
(D&E). But, unlike the other parts of the diagram, this outer circle primarily would be composed 
of existing dissemination and evaluation entities such as ERIC, technical assistance centers, 
Regional Educational Laboratories, organizations that provide third-party evaluations, National 
R&D Centers, publishers, participants in design competitions, and organizations managing ED 
and other related World Wide Web sites. Some of these D&E providers would specialize in 
specific topic areas and some in multiple topic areas. In some cases, national topic specific 
dissemination and evaluation organizations would be prime candidates to serve as support 
contractors for an expert panel in their topic area. For example, the Women’s Educational Equity 
Act (WEEA) Equity Resource Center is the support contractor for the Gender Equity Expert 
Panel. If needed, new topic specific dissemination and evaluation services could be created to 
support the work of expert panels in specific topic areas. It is likely that most D&E providers 
would also have key roles in other aspects of the NEDS and the more loosely configured R&D 
production system. Like the middle circle, participants in this outer circle would be linked across 
Panel topics by a governance structure 6 and by joint use of Special D&E Providers. Functions of 
topic specific and multi-topic Special D&E Providers would include disseminating promising 
and exemplary programs, obtaining and sharing systematic user feedback, reviews and evaluation 
reports on promising and exemplary programs in multiple topic areas and sharing information 
about opportunities to participate in the third-party evaluations of promising programs. In a 
commissioned paper, Patricia Campbell (1994) described initial ideas on how a D&E provider 
could obtain, synthesize and share teacher and student evaluation feedback on promising and 
exemplary programs. 

The spokes of the wheel show that much of the coordination of the work and contacts with the 
users would be organized in relation to each of the topic focused expert panels. However, there 
would also be many other connections since some dissemination and evaluation structures cover 
multiple topic areas and since promising and exemplary programs (i.e., gender equity in 
mathematics, science and technology) may be appropriately disseminated under the auspices of 
more than one topic area. 

Principle 1. The Federal Leadership Role Should be Facilitative. The federal government will 
play a facilitative leadership, partnership, community building role, more than a regulatory role. 
This new type of collaborative leadership with the field can be developed by establishing expert 



6 This governance structure may be part of NEDS or it may be created specifically for D&E 
' providers with ties to the expert panels. 
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panels in a wide variety of topic or special interest areas and sustained over many years with 
relatively modest federal investments because the topic-focused constituency groups would 
volunteer review and other services, much as they do now in reviewing journal articles. In 
addition to strengthening support for the agency from its concerned constituencies, this approach 
should help these topic focused communities develop a systematic process to share what works 
and identify gaps. The current panels are succeeding in this in two different ways. The Gender 
Equity Expert Panel has created an advisory group of over 1 00 experts who are willing to help 
the Expert Panel. And many of the members of the Mathematics and Science Expert Panel are 
leaders of key constituency groups in their area. This closer connection with constituency groups 
who have expertise in particular topic areas should also help make agency supported work in the 
area such as design competitions, and technical assistance activities more relevant to constituent 
needs. 7 

A facilitative federal coordination function is needed to help all components of the System learn 
from each other and work in complementary ways where there are natural overlaps in 
responsibilities. More specifically, it will: 1) provide leadership in coordinating (and where 
feasible, funding) all components of the Wheel Diagram, 2) develop systematic evaluation and 
dissemination methodologies and consistent decision rules and definitions for system activities 
so that the public will understand distinctions between “promising” and “exemplary” programs, 
and 3) develop incentives to make the System work. 

1) Coordination of components of the Wheel Diagram requires flexibility to take advantage of 
the interests of various topic focused federal offices and constituency groups. In doing so, it is 
assumed that federal programs in and outside of OERI would participate in this System, but that 
each would have somewhat unique activities to take advantage of their own legislative 
responsibilities. For example, in mathematics and science education, NSF might be able to 
support a D&E provider to manage comparative third-party evaluations of promising programs 
with similar purposes or they might fund a D&E provider to manage a design competition (as 
suggested by Robert Slavin, 1997) to fill a specific gap in the availability of exemplary replicable 
programs. ED might be able to support state professional development programs that choose 
promising and exemplary products and programs recommended by the Mathematics and Science 
Expert Panel or the Gender Equity Expert Panel. 



7 Constituency groups as represented by associations often recognize excellence by giving awards to 
individuals in their field or they help establish standards and credentialling systems which focus on 
judging the quality of individuals or organizations in meeting standards of the profession. They have 
established activities to recognize promising and exemplary programs, but often find it difficult to 
sustain this type of review effort without external support or clear connections to continued evaluation 
and dissemination efforts. The Department of Education has had similar experiences with a wide range 
of recognition and review activities. 
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2) Development of common evaluation and dissemination methods and definitions has started. 
The draft OERI standards for designating promising and exemplary programs have common 
criteria categories: 

- evidence of effectiveness/success (from self and third-party evaluations of performance) 

- quality (panels judge this by reviewing program materials, perhaps by observing the 
program in action. They may use criteria such as: congruency with research, accurate & up- 
to date content, promotes equity and is free of bias based on race, gender, age, culture, ethnic 
origin, disability or limited English proficiency, 

- educational significance (addresses important educational issues, has advantages over other 
programs with similar purposes) 

- usefulness to others (See Summary of OERI Standards and Criteria). 

3) Development of major System incentives revolves around consequences for receiving 
designations of promising or exemplary. One suggestion is to provide funds for improving and 
evaluating programs that are designated “promising”. This support would be geared to helping 
them qualify for future exemplary designations. Based on this understanding, a promising 
designation would encourage only cautious adaptation or adoption, during which time the 
program would undergo careful evaluation. Users would be encouraged to select exemplary 
rather than promising programs unless no exemplary programs meet their need, or unless they 
wish to participate in the field testing of promising programs. The key purpose for designating 
programs as exemplary is to increase the use of what works best. In addition to developing 
working relationships with specific D&E providers, the Expert Panels would work with others to 
publicize information on exemplary programs through the World Wide Web, TV and print media 
to help consumers choose from a variety of particularly worthwhile options. 

The active partnership roles for the topic-focused constituency groups will need to build on their 
strengths, resources and interests. For example, the Gender Equity Expert Panel has formed six 
subpanels to better connect with specific interest groups and the Mathematics and Science Expert 
Panel members have many contacts with mathematics and science educators they can tap through 
state affiliates of their national associations. 

Principle 2. There should be a combined focus on evaluation and dissemination which I call 
Consumer-Oriented Evaluation. This focus should be particularly appealing to educators who 
know they can do a better job if they are able to choose and use effective tools or instructional 
programs. This concept focuses on obtaining and disseminating descriptive and evaluative 
information designed to help consumers make decisions about what products or programs will be 
best for them. In addition to educators, consumers include the entire public (ranging from policy 
makers to students) who might be interested in learning about the merit of an education product, 
program or practice. Since many of these R&D-based tools have similar purposes, they will 
want to know their comparative merit on a range of criteria that are likely to be important for 
their own decision-making. The OERI standards categories as described in the sidebar are 
intended to provide a framework for topic-focused criteria and for research-based criteria 
selected by experts. Ideally, this System of Expert Panels should use D&E providers to develop 



OERI STANDARDS AND CRITERIA FOR EXPERT PANELS TO USE TO DISTINGUISH BETWEEN 
PROMISING AND EXEMPLARY PROGRAMS-summary based on 2/1 1/97 draft regulations revised 
according to public comment and suggestions from the OERI Board at their Jan. 31, 1997 Meeting in 
Washington, DC. A subsequent version of these draft regulations is to be published in the Federal Register in 
the spring of 1 997 for public comment. 

A panel may recommend to the Secretary that a program be designated as promising if the panel determines 
that the program is strong on each of the four categories of standards (Evidence of Success / Effectiveness, 
Quality of the program. Educational significance and Usefulness to others). A panel may recommend to the 
Secretary that a program be designated as exemplary if the panel determines that the program is excellent on 
each of the four categories of standards. The decision framework for determinations of strong and excellent on 
evidence of success/effectiveness is intended to be the same for all panels. However, on the other three 
standards categories of quality, significance and usefulness to others, each panel may add to the core criteria 
and establish its own decision framework for designation as promising and exemplary. It is expected that the 
panels will require a strong judgement on almost all criteria within each of these three categories for a 
designation as promising and a judgement of excellence on almost all criteria within each of the three 
categories for a designation as exemplary. For a program to be designated exemplary any weaknesses must be 
minimal and easily corrected. 

(a) Evidence of Effectiveness/ Success. 

To be judged strong in the category of evidence of effectiveness all of the following criteria must be met. A 
program must: 

• have defensible overall evidence supporting claims of worthwhile performance results (without 
substantial harmful results) at one or more sites (without failing at a large number of other sites) 

• have logical or other evidence of adaptability or transportability to other sites. 

The combination of this positive evidence of effectiveness and potential replicability creates a significant 
probability that the program will eventually be able to provide evidence to support claims of exemplary 
meritorious results as defined in the following section. 

To be judged excellent under the category of evidence of success/ effectiveness all of the following 
criteria must be met by convincing evidence that very important claims of positive results (or 
performance outcomes) can be: 

• reasonably attributed to the program, and 

• that evidence to support these claims of worthwhile results was sustained in multiple site 
replications within the past few years (without failing at a large number of other sites or being 
accompanied by harmful results). 

(b) Quality. The panels will make their judgments about quality by reviewing the program materials and 
determining the extent to which the program: 

(1) is congruent with sound research and practice 

(2) incorporates accurate and up-to-date information/content 

(3) promotes equity and is free of bias based on race, gender, age, culture, ethnic origin, disability, or 
limited English proficiency 

(4) is appropriate, engaging, and motivating for the intended audiences 

(5) contains materials that conform to accepted standards of technical product quality. 

(c) Educational Significance. The panels will use their expertise in the area to determine the extent to 
which the program: 

(1) addresses an important education issue, challenge or problem 

(2) has advantages over other programs with similar purposes 

(d) Usefulness to Others. The panels will make these judgments by using their knowledge of what is 
valued by educator and student users in determining the extent to which the program: 

(1) is reasonable in terms of costs to potential users in relation to expected benefits 

(2) is or can be made easily available to potential users 

(3) can be readily adopted or easily adapted in new locations 

(4) can be used in conjunction with other programs if appropriate 
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consumer reports 8 for specific topic areas where there are Expert Panels. These consumer reports 
could be supplemented by descriptions and summary reviews of individual promising and 
exemplary programs which a potential user could retrieve via a computer/Intemet search. 
Consumers should be able to use either or both of these information sources so they can do their 
own side-by side comparisons and make informed selection decisions. Based on past experience 
from Educational Products Information Exchange (EPIE) (Komoski, 1989) and others, it is 
unlikely that education consumers will want to pay for this information. So it is assumed that the 
government, foundations and associations will need to pay for D&E providers to do this public 
domain work. When all System components are operating, the evaluation results that would be 
included in this consumer information would not end with initial expert panel decisions. The 
panels and their support contractors would continue to up-date this information based on work of 
the D&E Providers described in the outside wheel of the System Diagram. For example, a 
Special D&E Provider would collect information from users of promising and exemplary 
programs, thus making sure that users play a continual role in helping each other learn what 
works for whom. It is assumed that this new feedback function and many of the nationwide 
operations of this System will become increasingly feasible as more educators use computers and 
Internet to communicate. 

Principle 3. This System should help create a deliberate, effective and continuous federal 
approach to finding and sharing the best and identifying gaps which merit R&D support. 

Many federal education programs fund “demonstration” projects which generally allow for the 
development of a creative program in one site. But when federal funding ends the program may 
also end in the original site and there are few opportunities to learn if it merits continued support 
for additional revision, evaluation or dissemination to others. Similarly, federal offices rarely 
provide this type of additional support for meritorious programs not developed with federal 
funds. Thus, the System of Expert Panels is intended to carefully identify the gaps and 
opportunities and may help target subsequent federal funding to replicable programs with the 
greatest chance of helping students receive a better education and show improved performance. 
While most Expert Panels will be designed to cover the broad interests of their constituency 
groups and to last over a number of years, it is also possible for more short-term specific focused 
expert panels to be established and funded by a federal office or for a broader panel to limit their 
search for solutions to priority areas during specific time periods (Datta & Scriven, 1997). 

Principle 4. National dissemination efforts should find multiple ways to encourage selection 
and use of the best education R&D has to offer. 

This System is not based only on recognition and awards for excellence. Instead it focuses on 
learning about or developing what is likely to be useful to others. The assumption is that most 
users will find the funds for implementation especially of exemplary models or designs in late 



8 

Consumer Reports from the Consumers Union could be one model. Each report would contain an 
overview describing the programs and key issues related to their merit, a comparison chart and individual 
program summaries. 
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stages of development. If programs are designated as exemplary by the System of Expert Panels 
the federal government could encourage recipients of Title I or other federal funds to choose 
from among the exemplary programs or provide technical assistance or other support for their 
implementation. Incentives could also be provided to collect evaluation evidence on promising 
and exemplary programs to learn more about how they work in different situations. This 
evidence could be collected, analyzed and shared by a D&E provider. 

Principle 5. All participants will increase their commitment to using research, development 
and evaluation to learn what works well to inform and foster revision (adaptation) and 
replication and thus improve education. Many recognize the need to help schools obtain good 
evidence on the effectiveness of replicable models so they will have justification for choosing 
among them or sticking with their current practices. 

Principle 6. The System of Expert Panels is very ambitious and likely to face challenges such 
as: 

- Obtaining major financial support for individual panels and for related D&E providers. 

- The replication of what works focus may also be opposed by educators who believe that 
replication of model programs and products is not a feasible strategy for educational 
improvement because of the need for substantial adaptation 9 or because they think educators 
must develop their own approaches based on their interpretations of research findings and 
their own insights and experiences. 

- Technical and procedural challenges related to deciding on the promising or exemplary 
status of submissions to the expert panels are substantial. Examples include difficulty in 
agreeing on desired performance indicators, frequent lack of information on adequacy of 
implementation as well as performance outcomes, and practical difficulties in making sure 
that evidence collection and panel submissions will be “the whole truth and nothing but the 
truth”. Although the System is designed to judge the program, not the developer, it is 
difficult to separate developers from their programs especially when the evaluations are 
likely to be dependent on the cooperation of the developers and thus they would not be 
anonymous. 

Conclusion: 

The other presentations in this session and the experiences of the pilot expert panels should help 
us learn how to build on the work of others to make this System a basis for the U.S. Department 
of Education to develop a systematic integrated leadership approach toward working with R&D 
and practitioner experts in a wide variety of topic areas. The symposium 1 6.23 tomorrow 
afternoon on the Gender Equity Expert Panel will show how important it is for experts in R&D 



All involved with the System of Expen Panels agree that adaptation is a fact of life, but that it often helps to start with 
something that has worked for others. 
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to work together to learn what works best in their areas of special interest. (I thank both the 
AERA Special Interest Group: Research Utilization and the AERA Women’s Committee and the 
Special Interest Group: Research on Women in Education for sponsoring these sessions.) 
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